ࡱ> vxu @ bbjbj)) .KzKzY%8t,,%dd"eeeI%K%K%K%K%K%K%$&R)~o%aeo%%I%I%!"X `<W"5%%0%q"* *4""v*#4edIeeeo%o% \$X\THE IMPACT OF RESEARCH ON EDUCATIONAL POLICY: AN EXAMINATION OF RESEARCH INTO STANDARDISED ASSESSMENT TASKS Gina Donaldson Christchurch University College, Canterbury, UK Abstract The introduction of the National Curriculum 1988 led to a regime of national testing in the form of Standardised Assessment Tasks (SATs) at the end of Key Stages 1, 2 and 3. The results of these tests were, and still are, used to create hierarchical league tables of schools. Research by Cooper and Dunne (2000) critiqued the effectiveness of one aspect of the SATs for mathematics. This paper will briefly summarise the researchs aims and findings. It will suggest that there are underpinning elements of positivist, neo realist and reductionist values within the testing policy. It will discuss the challenge the research presents to these values. Atkinsons work (2000b) will be explored as a useful framework for an analysis of the relationship between the research and the current policy. Aspects of the general relationship between research and policy which are exemplified by this specific case, and issues relating to the social justice role of research and processes of policy making and amending will be discussed. It will be concluded that it is the function of research to question policy but that the relationship between researchers and policy makers is complex, and in this case the response of policy makers does not answer the questions raised by the research. Key Concepts: Standardised Assessment Tasks, Realistic mathematical problems, Research in educational policy Introduction The use of SATs to assess childrens knowledge and understanding forms the basis of public comparisons of children and schools. The premise is that these tests must be a reliable and valid form of assessment. However, research by Cooper and Dunne (2000) questions this assumption. Furthermore, it suggests that written SATs are particularly inaccurate with certain groups of children who are thus disadvantaged by the policy of testing and ranking. This paper will firstly consider the research by Cooper and Dunne (2000) before going on to briefly discussing the underpinning values of the policy and the challenge presented to them by the findings of the research identified. The research findings The research is set in the context of politically driven reform of curriculum content and assessment, and an extensive programme of national testing. The results of this testing is used by the media, parents and the government to compare, and to name and shame schools. Schools, teachers and children bear the brunt of the resulting criticism. However, there is little critique of the tests themselves. Little notice in the past has been given to contextual factors and their impact on attainment. Performance in SATs has been seen as more valuable than teachers assessment of children across the curriculum throughout a year (known as Teacher Assessment.) The research aims to question the way in which SATs reflect what children actually know and understand. In particular, the ability of the SATs in measuring childrens understanding of realistic word problems is explored. Realistic word problems are included in the curriculum as part of Attainment Target One for Mathematics, Using and Applying Mathematics. This is seen as important to motivate children and to provide learning which will be useful in childrens later lives through application of more abstract knowledge and skills. For example, an esoteric or context free problem might be 15 divided by 6. A realistic problem which is designed to apply the same mathematical knowledge might be how many egg boxes are needed to carry 15 eggs. The problems are called realistic because they are designed to reflect everyday mathematical knowledge and the sort of context where mathematics may be used. Their solution should require an element of mathematical thinking. However, the problems are clearly not real themselves. Data were collated on children in the 10 to 11 and 13 to 14 age groups, at the end of Key Stages 2 and 3. Test performance data for these children were collected using test papers which included National Curriculum SATs questions which were both realistic and esoteric, or context free. The children were also interviewed and asked to describe how they approached the solution of a number of the realistic items in the tests. They were asked to describe their approaches and the reasons for their choice of method. In this way, the measure of the childrens understanding through the test was compared to the understanding and knowledge evident in the interview. The childrens responses were often much more complex than the test item and mark scheme recognised. The research found that certain groups of children perform poorly on realistic items within tests because they are less knowledgeable about the peculiar ways boundaries are drawn between school mathematics and mathematics in everyday life. This is not recognised by the official mark scheme, so these children fail to demonstrate what they understand of mathematics when solving such problems. The boundaries between school mathematics and mathematics in everyday life can be identified in the following question: This is the sign in a lift at an office block. This lift can carry up to 14 people. In the morning rush, 269 people want to go up in this lift. How many times must it go up? Schools Examination and Assessment Council (SEAC) 1992 The marking scheme (SEAC 1992) provides guidance on what is appropriate evidence of understanding. Gives the answer to the division 269 by 14 as 20. Do not accept 19 or 19.2 In responding to the realistic problem, children need to identify the mathematical operation of 269 divided by 14 and then apply enough real life knowledge to recognise that their answer must be rounded up to a whole number to correspond to the number of times the lift has to be used. The opportunity of using a calculator in the paper means that a full decimal answer can be easily obtained. However the children must not apply too much real life knowledge to consider other factors. For example, what if the lift is not always full, if some people decide to walk or some take up more space than is expected because say of wheel chairs? Too much real life knowledge would lead to an inappropriate response. The children are required, unintentionally it seems by the test design, to negotiate exactly how much real life knowledge to apply to the problem. The authors are rightly sceptical about their conclusions especially where they relate to social class. However, the research finds quite convincingly that at the end of Key Stage 2, children from what they describe as working and intermediate class backgrounds perform less well than those from service class backgrounds on realistic items as compared to esoteric items. There are smaller differences relating to gender, with boys performing better than girls on realistic items. In their exploration of why children from working class backgrounds found the conventions of expected responses to realistic problems more difficult to negotiate, the authors draw on both Bernstein (1996) and Bourdieus (1986) accounts of social class differences in cultural competencies p67. They suggest that the use of SATs with simple mark schemes in these cases did not provide a valid test of all childrens mathematical knowledge, and therefore their findings challenge the status of SATS as a measure of learning. What are the underpinning values of the policy of testing against National Curriculum Attainment Targets? The policy of a statutory National Curriculum enforced by testing suggests that it is possible and useful to identify valuable knowledge and skills, and then measure learning of these through written tests. The knowledge, skills and attitudes encapsulated in the National Curriculum are offered as a definition of what is valuable in the primary curriculum and the regime of testing is an unquestioned means of measuring learning. This emphasis on measurable outcomes corresponds to a positivist view (Pring 2004) rather than a constructivist belief. There are also elements of a neo realist view within the policys emphasis on testing. The use of tests relates to the belief that there exists a real world independent of our interest or knowledge of it (Furlong 2004 p347). It seems reasonable to suggest that mathematical ability exists independently. However the policy is based on the belief that SATs provide a valid measure of mathematical ability, and that the mark scheme defines that ability accurately and comprehensively. Scott et al (2002) suggest that in the neo realist view, all possible accounts are not equally valid, credible or legitimate p297. Therefore the assessment of childrens learning by teacher assessment is not of equal value as the assessment by SATs, even though the teacher assesses in a variety of contexts over a year and across the whole mathematic curriculum. The policy therefore presents a fixed curriculum and tool for assessment. Radford (2006) in his consideration of complexity and chaos inherent in education, explores what he says is a common reductionist view of education. The current policy closely ties its definition of effective education to performance in SATs. Educational situations and events are assumed to comprise of particular elements working together in a linear and casual relationship (Radford 2006 p.180). The SATs themselves he says, as part of the school effectiveness paradigm, are too often treated as unproblematic p181. However it seems that the relationship between the worth and value of childrens learning in the primary phase and their performance in SATs is so close that the two become indistinct. How does the research challenge the underlying values of the policy? The research challenges the idea that education lends itself to measurement through written tests with a simple mark scheme. The findings of Cooper and Dunnes research question the validity of realistic problems presented in the SATs to measure all childrens learning. Childrens responses to realistic problems, it is suggested, can not be accurately measured in this disinterested, scientific way. The complexity of childrens responses to the realistic problems revealed by Cooper and Dunnes research endorses Radfords picture of education as characterised by complexity. The responses reveal the difficulties when mathematical ability is reduced to a SATs level, when mathematics itself is reduced to written test questions and the definition of mathematical ability is reduced to a simple mark scheme. The suitability of education as a subject for scientific measurement has been widely discussed (eg Furlong 2004, Radford 2006). Berliner (2002) suggests that to argue that science itself is a simple process of hypotheses testing is too simple. He states that science is to do with unrestricted questioning and the ignorance of authority p1. This view of science recognises the complexity of education, for example, in the many contexts and interactions involved in any one episode in a classroom. Others such as Bassey (2001) have suggested that education is characterised as involving many variables and research provides little data on these. Research on education can therefore only lead to fuzzy generalisations. The idea of fuzziness he takes from Kosko (1994) as being multivalence, analogue rather than binary, infinite shades of grey between black and white p18.The idea then that SATs can provide an accurate picture of childrens learning must be questioned and Cooper and Dunnes research demonstrates this is so. Atkinson (2000b) provides a useful framework for analysis of the challenge the research presents to the policy and its underpinning values. In her exploration of the impact of postmodernist thinking on educational research, she explores the idea of education as a fixed concept of what works and a questionable certainty about best practice ( Atkinson 200b p.81) which in this case is demonstrated by the policy makers. She states that postmodernist thinking allows a broadening of the field of educational research and policy to embrace multiple voices and persepectives (ibid. p.83). This might run the risk of fragmentation of views but allows debate and a useful element of uncertainty. Where Cooper and Dunnes research might not claim to be postmodernist, these ideas offer a useful analysis of its challenge to the policy. Atkinson identifies four key areas where research can challenge centralised control. Firstly the research challenges the certainty of authority of knowledge and therefore power. Cooper and Dunnes findings question the knowledge presented in the National Curriculum as it is interpreted into realistic problems and in particular the design of test questions and their mark schemes. Atkinson explores the idea that this power/knowledge relationship results in privileging one term, concept and approach over others. Certainly the policy has led to a comparison of children, schools and local authorities and thus a clear hierarchy. Cooper and Dunnes research questions the validity of this. Secondly research based on post modernist thinking challenges the certainty of the authority of a specific mode of learning. It questions the idea that childrens learning of the knowledge, skills and processes within the National Curriculum is linear, structured, stable and controllable (ibid. p.87). The National Curriculum itself does not sit comfortably with other views of education such as Dewey (Pring 2004). Post modernist thinking would describe learning as partial, unstable and contingent (Atkinson 2000b p.91). Cooper and Dunnes research goes some way to explore the complexity of learning of mathematics in realistic contexts and the model of assessment by written tests. Thirdly Cooper and Dunnes research could be said to challenge the certainty of authority of identity. Atkinson states that end of Key Stage tests are an example of the result of policies which create textual identities (ibid. p.92). These are normative identities which create an image of a child or school. The image presented by the failure to reach level four at the end of Key Stage 2 is very real to teachers, governors, parents and children themselves. Children strive to reach level 4 and schools are driven to enable them to do so. Post modernist research, Atkinson claims, provides a different reading of the textual image. Cooper and Dunnes research provides an alternative reading of the response of children to realistic problems within the SATs. Finally Atkinson identifies the role of research based on post modernist thinking in challenging the certainty of authority of language. The end of Key Stage assessments bring their own terms and meanings: disapplication, springboard, booster etc. They also bring textual silences which do not speak of other voices, other identities, other ways of knowing (ibid. p.93). These other ways of knowing, identified by Cooper and Dunne, are not valid as they are too complex to be captured within the mark schemes of the SATs and therefore are rejected by the policy. What can be concluded generally about the relationship between policy and research? Humes and Bryce (2001) in exploring the relationship between scholarship, research, policy and practice, define one of the desirable characteristics of scholarship as academic freedom. This involves, they state, a willingness to question orthodoxies. The scholar researcher, we will suggest, has an important function as a public intellectual with responsibility to contribute to debate on matters of public policy. That contribution will take account of the agendas and priorities of government but will not be determined by them (Humes and Bryce 2001 p.330.) Cooper and Dunnes work seems to aim to fulfil a social justice function within this role of questioner of policy. The research highlights the plight of those in society most at risk of failing to reach the required level in the end of Key Stage assessments. Therefore the function of research can be seen as one of ensuring social justice for all members of society in the light of particular policies. Humes and Bryce go on to discuss the complex relationship between scholar researcher and policy maker. They highlight the tension within this relationship due to the opposing drives for academic freedom and centralised control. This is often complicated by funding issues. Researchers and policy makers have opposing ideologies, languages and goals. This is more evident when research is on policy itself. Furthermore we know from the literature on policy making that even where the research evidence is clear and often it is not the various stages of process (from conception through consultation, development and implementation) are often far from logical. Policy making is messy, involving political intrigue and compromise, subject to professional and bureaucratic self interest, and beset with operational difficulties (Humes and Bryce 2001 p.343). A full response to this research required considerable policy amendment or an admittance of policy failure. This is particularly difficult for a government as it demonstrates plainly a change in values and weakness in the initial policy. To radically change a policy endorsed by the same political party is more difficult than creating a new policy. The end of Key Stage One SATs are now optional, the timings for Key Stage Two assessments are less rigid, and league tables for 2006 include contextual value added ratings for schools. Does this represent a change in values or a superficial change in procedures? The rationale is not clear. Policy makers are under pressure to present initiatives to teachers as not only valuable but as quick and easy to initiate, particularly in a current climate of concern for teachers work load. A radical change in the assessment arrangements or complete move away from national testing might appease teachers principles but runs the risk of angering teachers who have spent many hours on their preparation, administration and evaluation. It seems that policy makers decisions need to be based on more than just research findings. Hammersley (2003) argues that the role of research is simply to provide a framework within which policy decisions can be made. However, Humes and Bryce talk of policy communities. Although these are not defined precisely, Humes and Bryce state that the communities should be intellectually engaged and characterised by questioning of orthodox practices. Clearly researchers should, in their view, be included in such communities and it is expected that the meetings of these communities be fiery and challenging. Conclusions The findings of Cooper and Dunnes research (2000) lead to conclusions which, although tentative, nevertheless challenge the policy of national testing. This is clearly an important role of research, to question policy. Mortimore (2000) states that it is the function of research to say that the king has no clothes (Mortimore 2000 p.18) or to be troublesome (Furlong 2004.) Ozga (2000) states that research into policy is essential, and must not be reduced to research for policy. Inevitably, research into policy causes tension, however valuable such tension may be. It is difficult for policy makers themselves to critique fully the effects of policies; those who are close to a particular development are not in a good position to recognise its problems. To do so requires the systematic and critical but nonauthoritative - eye that research can provide (Mortimore 2000 p.18). This, Mortimore asserts, is the duty of researchers, whilst maintaining their integrity. Realistic items with simple mark schemes have remained part of the SATs, despite the findings of the research. Concerns over the time and work required by the SATs from teachers have restricted test design. Recently though, there has been a move away from an assertion that a rigid administration of the SATs is the only valid and reliable assessment of childrens learning at the end of a Key Stage (Brooks and Tough 2006) However, changes to the policy seem to be largely procedural in nature, showing flexibility in order to lessen demands on children and teachers. It is clearly difficult for policy makers to justify fundamental changes to existing policies implemented or openly supported by their own party, given the concern over teachers work load and the need for the government to appear firm and secure in its beliefs. It seems that the processes of policy making and change are complex themselves. The research remains largely unrecognised and its questions left unanswered. Bibliography . Atkinson, E. (2000b) The promise of uncertainty: education, postmodernism and the politics of possibility, International Studies in Sociology of Education, Vol. 10, (1), pp. 81-99. Bassey, M. (2001) A Solution to the Problem of Generalisation in Educational Research: fuzzy predicition Oxford Review of Education Vol 27 (1), pp. 5 2. Berliner, D. (2002) Educational Research: The Hardest Science of all Educational Researcher Vol 31 (8), pp. 18 20. Bernstein, B. (1996) Pedagogy, Symbolic control and Identity: Theory, research, critique London: Taylor and Francis Bourdieu, P. (1986) Distinction: A Social Critique of the Judgment of Taste London: RKP Brooks, R and Tough, S. (2006) Assessment and Testing: making space for teaching and learning London: Institute for Public Policy Research Cooper, B. Dunne, M. (2000) Assessing Childrens Mathematical Knowledge Buckingham: OUP Furlong, J. (2004) BERA at 30 Have we come of age? British Educational Research Journal Vol 30, (3), pp. 343 358. Hammersley, M. (2003) Can and should educational research be Educative? Oxford Review of Education Vol 29, (1), pp. 3-25 Humes, W. Bryce, T. (2001) Scholarship, Research and the Evidential Bases of Policy Development in Education British Journal of Educational Studies Vol 49 (3) pp. 329-352. Kosko, B. (1994) Fuzzy Thinking London: Harper Collins Mortimore, P. (2000). Does educational research matter? British Educational Research Journal Vol 26 (1), pp 5 24. Ozga, J. (2000) Policy Research in Educational Settings Buckingham: OUP Pring, R. (2004) Philosophy of Educational Research (Second edition) London: Continuum Radford,M. (2006) Researching Classrooms; Complexity and Chaos British Educational Research Journal Vol 32 (2), pp. 177 190. Schools Examinations and Assessment Council (SEAC) (1992) Mathematics Tests 1992 London: SEAC /University of London Scott, D. Simons, H. Usher, R Pring, R. (2002) Fussing about the Nature of Educational Research: the neo realists versus the relativists British Educational Research Journal Vol 28 (2), pp. 291 296.     PAGE  PAGE 1 -.klm| j | }    [  Ƕ|unungnugugucucuc_uXc_ h# 3h#&h5h{l h# 3hn h# 3h h# 3h^h# 3h^5 h<5 h)n75h)n7h/OJQJ^Jh)n7h)n7OJQJ]^Jh)n7h)n7OJQJ^J h<h<CJOJQJ^JaJ h/5#h<h/5CJOJQJ^JaJh<5CJOJQJ^JaJ#h<h<5CJOJQJ^JaJ".lm| IJWgd<gdII$a$gd)n7 $dha$gd)n7 $dha$gd<dhgdII $dha$gdIIab  ! e g x    6 < = @  Z t u {  (*IJWce/9=>`a»ѷѷѰ h# 3h h# 3hr hbh5h# 3hI"5 h# 3hOFh< h<h<h<hOF5 h# 3h{tf h# 3hnhF h# 3hKfh{tf h# 3h#& h# 3h h# 3h^h{l9a";o{#(H}0sĽĶįĶĶĶħĶĠīĶĶĶħć h# 3h ja h# 3hs h# 3h )hF h# 3h hh{tf h# 3h h# 3hk h# 3h^ h# 3h_<h# 3h5h# 3h95h# 3hb5hI" h# 3hKPq h# 3h h# 3hr hb5!B7@IMVWhpB`f35AZ@A^_ !!M"O"P"z###ݿݿݸݸƪƪƣʣƪ h# 3hC h# 3h h# 3ho h# 3hKo h# 3hshChFh9 h# 3hh h# 3hs h# 3hY h# 3hk h# 3h ja h# 3h )@ZA!%%%(*p+,,8-N-O-...159;>A dhgdIIdhgdII#####$$$$$$$$ % %%%)%2%%%%%%Q&\&h&&&&''','-'''''''(([((()))**++o+p++++, , ,žžžŷŬŨšš h# 3h ja h# 3h|mh|hZ 5 h# 3h| h# 3h] h# 3h)< h# 3h1h# 3h)<5h# 3h15 h# 3hKohF h# 3hH~ h# 3hC h# 3hhC< ,O,,,7-8-9-F-M-O---...../$/)/C/E/T/q/z////:0 1 1120313333Z4{4_5`555e66T7d7e7f7h7i77-8žźźžœœœŌ h# 3hr/3h. h# 3hj(1 h# 3heP h# 3h)< h# 3h S` h# 3hM`ehI h# 3h] h# 3h 9h# 3hj(15h# 3hM`e5 h# 3h9hFhKb h# 3h ja h# 3h[W5 h# 3h|m5-8.84858<8D8E8F8H8I899:G:::d;;;;;A<<<======= =!=#========>V>>>> ? ?-????????@@,@s@GATAAAܼѼѣѪʣ h# 3h% h# 3hI# h# 3h jahP h# 3hWG h# 3h 9 h# 3h] h# 3h)<hqz h# 3hx h# 3h8h.hI h# 3hr/3 h# 3hzbr?AAAAABJBNBnBCC CCCCCC4CDCQCRCUC`CCCCCCC,DADDD'E)ESETEEECFDFYFZF[F\F`FaFcFeFGGsHHHIII9K:Küզܢբբܦ՛딭ܔ h# 3ha h# 3hA-hKb h# 3hm h# 3hDHh# 3h+5 h# 3hzbr h# 3h% h# 3h)<h. h# 3h+ h# 3h jahPh. h# 3h] h# 3h3 h# 3h#:AJBCC,D(E)EDFdFeFGIIXKYKQQQTTcUzU{UUYYYYdhgdII:KOKVKWKYKKKKLLMNNNOOOOOOmPnPwPPQQQQJRKR^R_R;Sd>e>?AAXCYCIIILLcMzM{MMQQQQyRSSUTT8UU'VV WWWXXYYYYYYYYYYYYYYYYYZ0000000000000000000(00p0p0p0p0p0p0p0p0p0p0p0p0p0p0p0p0p0p00p0 0p0p0p0p0p0p0p0p0p00p0p0p0p0p0p000p00p00p0p0p0p0p0p0p0p0p0p0p00p0p0p0000P0p0p0p0p0p0000p0p0p0@0@00@0@00@0@00@0@00@0@0@0@0@0@0@00 .lm|IJWZ000000h000000000 0 0t a $$$' a# ,-8A:KSYU\^ab25679:;<>?@BCDAYab38=AEb4  '!!8@0(  B S  ?ZM"MMJ>??:C?CwH|HHHRRRRRSpSsSUUVV'V,VVVVVhWkWX'XXXYYYYYYYYYYYYYYYZ^c& 8%M%},,T/^/J:O:D>a>JvKxKKLLcMyM\RbRyRSUUVV%X'X@YGYYYYYYYYYYYYYYZ3333333333333333333kmm| (* 8%O%):J:D>e>IIcM{MQQU'VYYYYYYYYYYYYYYZYYYYYYYYYYYYYZ | x}xFG~h,P|xx28!Z?@ABCDEFHIJKLMNPQRSTUVWXYZ[\]^_`abcdfghijklnopqrstwRoot Entry F@ yData G1TableO*WordDocument.SummaryInformation(eDocumentSummaryInformation8mCompObjj  FMicrosoft Word Document MSWordDocWord.Document.89q